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Abstract 



A system, methods, and algorithm for content-based image retrieval and recognition system, 
useful in all types images and image formats. An image(s) or an image segment(s), which is 
specified by the user in two clicks (the first in the upper-lefl comer and the second in the bottom- 
right comer), specifies the content-based sample. The sample image(s) is used to teach the 
system what to look for via the ABM (Attrasofl Boltzmann Machine) algorithm and APN 
(Attrasofl PolyNet) algorithm; the system then searches through one or many directories, which 
is specified by the user, and presents the search r e s e arch results. The search result consists of 
pairs matched image and a Weight (score), which specifies the similarity between the sample and 
matching images. These weights are also being used to classify images in the cases of the 
classification problem. The users are able to view the retrieved images in the result via a single 
click. When the algorithm is implemented as a software component, the system integration will 
follow the specification of the "Attrasoft Image Verification and Identification Application 
Programming Interface (IVI-API)". 
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TECHNICAL FIELD 



The present invention relates generally to image retrieval and image recognition, and more 
particularly related to a system, methods, and algorithms of content-based image retrieval and 
recognition system. Within such a system, the image(s) to be retrieved/recognized is not 
preprocessed with the association of key words (meta-data). This system allows the user of an 
image retrieval/recognition system, such as software together with a computer, network server, 
or web server etc, to define a searching criteria by using an image(s), a segment of an image(s), a 
directory containing images or combinations of the above. This system will retum the result, 
which contains pairs of the matched image and similarity. The user can see the matched images 
in a single click. 

This invention can be used in image verification (1-to-l matching, binarv output: match/no 
match 1:1 matching, binary output: y e s/no ), image identification (1-to-manv matching, single 
output to indicate a classification 1 :N matching, singl e output to indicat e a olassifioation) , image 
search or retrieval (l-to-manv matching, multiple output 1:N matching, multipl e output ), and 
image classification (many-to-l or many- to-many matching N:l or N:N matching ). For 
simplicity, we will only use the word, retrieval. 
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BACKGROUND OF THE INVENTION 



In certain types of content-based images retrieval/recognition systems, the central task of the 
management system is to retrieve images that meet some specified constraints. 

Most image-retrieval methods are limited to the keyword-based approach. In this approach, keywords and 
the images together form a record in a table. The retrieval is based on the keywords in much the same 
way as the relational database. (Example: Microsoft Access). 

The user operation is generally divided into two phases: the leaming phase and the search/recognition 
phase. In the leaming phase, various types of processes, such as image preprocessing and image filtering 
are applied to the images. Then the images are sent to a recognition module to teach the module the 
characteristics of the image. The leaming module can use various algorithms to learn the sample image. 
In the search/retrieval phase, the recognition module decides the classification of an image in a search* 
directory or a search database. 

A very small number of commercially available products exist which perform content-based 
image retrieval 

Informix Internet Foundation.2000 is an object-relational database management system 
(ORDBMS), which supports non-alphanumeric data types (objects). IIF2000 supports several 
DataBlade modules including the Excalibur Image DataBlade module to extend its retrieval 
capabilities. DataBlade modules are server extensions that are integrated into the core of the 
database engine. The Excalibur Image DataBlade is based on technology from Excalibur 
Technologies Corporation, and is co-developed and co-supported by Informix and Excalibur. 
The core of the DataBlade is the Excalibur Visual retrievalWare SDK. The Image DataBlade 
module provides image storage, retrieval, and featxire management for digital image data. This 
includes image manipulation, I/O routines, and feature extraction to store and retrieve images by 
their visual contents. An Informix database can be queried by aspect ratio, brightness, global 
colour, local colour, shape, and texture attributes. An evaluation copy of IIF2000 and the 
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Excalibur Image DataBlade module can be downloaded from Informix 
www.informix.com/ e valuat e / . 

IMatch is a content-based image retrieval system developed for the Windows operating system. 
The software was developed by Mario M. Westphal and is available under a shareware license. 
IMatch can query an image database by the following matching features: colour similarity, 
colour and shape (Quick), colour and shape (Fuzzy), colour percentage, and colour distribution. 
A fully functional 30-day evaluation copy is available for users to assess the software's 
capabilities and can be downloaded from Mario M. Westph al's web site 
www.mwlabs.do/download.htm The shareware version has a 2000 limit on the number of images 
that can be added to a database. A new version of the software was released on the 18th February 
2001. 

The OracleS/ Enterprise Server is an object relational database management system that includes 
integral support for BLOBs. This provides the basis for adding complex objects, such as digital 
images, to Oracle databases. The Enterprise release of the Oracle database server includes the 
Visual Information retrieval (VIR) data cartridge developed by Virage Inc. OVIR is an extension 
to OracleSz Enterprise Server that provides image storage, content-based retrieval, and format 
conversion capabilities through an object type. An Oracle database can be queried by global 
color, local color, shape, and texture attributes. An evaluation copy of the OracleSz Enterprise 
Server can be downloaded from Oracle otn.oracl e .oom . 



6 



TECHNICAL BACKGROUND 

The concepts in this section are well known and they are not a part of this invention. Thev are 
listed to facilitate the understanding of the rest of this invention because these concents are 
generallv not grouped together in anv textbooks or publications. In particular, the various spaces, 
distances in these spaces, and their relationship with image recognition are explained. 

1. Images 

An image consists of a set of pixels. For example, a 480x640 image will have 307,200 pixels. 

A pixel can have a set of values. For example, a pixel in a red color image can have three values: 
RGB = (Red, Green, Blue) = (255. 0, 0). A grev image can reduce three numbers into a single 
number. For example, a grev pixel has a value =128. For simpUcitv, in the rest of this section, 
we assume each pixel has a single value. 

There are two wavs to describe an image. Each description vields different features of the image. 
We will first introduce these descriptions: we will then explain the image spaces and the 
distances between images in these image spaces. 

The first way to describe an image is to use a vector. For example, a 5-pixel image is described 
as (10, 20, 30, 40, 50). A vector is also viewed as a point in a vector space: therefore an image 
can be viewed as a point in a vector space. For example, a 480x640 image can be viewed as a 
point in a vector space of 307,200 dimensions. 
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The second way to describe an image is to use a set of particles. First of all a binary image 
(black-and-white) is described by a set. For example, a S-pixel image is described as 10101. Let 
the pixels be described by five elements in a set: 0, L 2. 3. and 4. then the binary image 10101 
represents the set (0. 2, 41. Here is another example. 11100 represents the set 10. L 2). N 
elements will yield 2^ subsets. N pixels will yield 2^ black-and-white images. All of the 2^ 
subsets together form a space. A subset is a point in this space. All of the 2^ black-and-white 
images together form a space. A black-and-white image is a point in this space. Now we extend 
this definition of images beyond the binary values. Consider each element in a set is a mass 
particle that has an integer mass value, then a color image can be described as a set of mass 
particles. For example. (128. 0. 256. 0. 128) is a set of mass particles (0. 2. 4). where particle 0 
has mass 128. particle 2 has mass 256. and particle 4 has mass 128. 

To summarize, an image is a point in a space. The space can be a vector space, which consists of 
a set of vectors. The space can be a subset space, which consists of a set of subsets. 

In either description, the distance between two points can be defined, i.e. the distance between 

two images can be defined. In a vector space, the distances are -distances, where p = 1. 2 

00. For simplicity, consider a two-dimensional vector space, then the distances are: 

Li^= Ix^-Xjl + I y2-yi I 
L2 = V(x2-xi)^ + (v2-yi)^ 
U; = max I Ixj-x^l . I y^-Vj I I 
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In a subset space, the distance between two points is called the Hausdorff distance. We will only 
use an example to illustrate what the Hausdorff distance is. Consider two cities as two sets, the 
Hausdorff distance between the two cities is this: vou can start from any point in one city, after 
traveling the Hausdorff distance toward the second city, you will be guaranteed to be in the 
second city. 

2. Mapping 

Mapping establishes a relationship between a Domain to a Range. Both a Domain and a Range 
are sets. For example, a Domain can be la, b) and a Range can be 10, 1), then a mapping is (aO, 
bU. If both Domain and Range are a set of finite elements, when the Domain is removed, the 
Range will form a vector. In the above example, when the domain is removed. The Range is a 
vector, (0, 1). 

3- Markov Chain and Pattern Classification 

A Markov chain is a sequence of random values whose probabilities at a time interval depends 
upon the value of the number at the previous time. The controlling factor in a Markov chain is 
the transition-probability; it is a conditional probability for the system to go to a particular new 
state, given the current state of the system. The key feature of a Markov chain is its invariant 
distribution: after a Markov chain evolves long enough, the probability for each configuration to 
appear is fixed. 

For example, let a Markov chain have two states, 0 and 1. The transition matrix elements are, t^ 
= tj n = 0.5, t_m = 0, t_M = 1. A possible evolution is 0001 11111... Another possible evolution is 
00000001 1 1 1 1 1 1 1. The invariant distribution is f(0) = 0, fd) = 1. 
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This invariant distribution can be used to classify patterns as follows. Let x be an image, and let 
a, b be two classes. We can construct the two possible vectors: (x, a) and (x, h\ Let an invariant 
distribution function of a Markov chain be z = F (\\ where v is a vector. Assume v = (x. a), z = 
zl: and v = (x. b), z = z2, then this invariant distribution function actually classifies the image, x. 
as follows: the probability of x in class a is zl: and the probabilitv of x in class b is z2. If we can 
further assume zl » z2, then x is classified as a member in class a. 

In particular, assume a Markov chain is constructed with an image p. Let x be an image, and let 0 
(No Match), 1 (Match) be two classes: then the two possible vectors are (x, 0) and (x, 1). After 
the Markov chain evolves long enough, an invariant distribution of the Markov chain is reached. 
Let a distribution function be z = F (v), where v is a vector. If v = (x, 0), z = zO: and v = (x, 1), z 
= zl, then the probabilitv of x in class 0 is zO and the probabilitv of x in class 1 is zl. The 
matching score between image p and image x will be zl. 

A triplet is (x, 0 zO) or (x, 1, zl). A doublet is (x, zl). 

4. Markov Chain and Neural Network 

A particular tvpe of neural network is called a Boltzmann Machine, in which each neuron has a 
certain probability to be in state 0 and a certain probabilitv to be in state 1. A Boltzmann 
Machine forms a Markov chain. 

5. Artificial Neural Net 

The artificial neural net is a dynamical system, which will transit from one state to the next 
according to its internal connections. 
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The configuration space is set of all possible configurations. For example, assume there are 

three binary neurons and the set of all possible configurations is (OOP. OOL 010, Oil 111). 

Each configuration in this space is considered as a point. In general if there are N neurons, there 
will be 2^ points in the configuration space. 

An input image will ultimately be mapped into a neural net. The number of pixels, in general 
does not agree with the number of neurons. Mapping an input image into a given neural net will 
be necessary. 

The connection space in this invention is the set of all possible connections. For example, 
assume there are three binary neurons ( 0. h 21 and the set of all possible connections is 
{nnthin p. 0. L 2. PL 02. 12. 012 where (0. 1, 2) are self connections. (01. 02. 121 are regular 
connections, and (0121 is a high order connection. Each connection in this space is considered as 
a point. In general if there are N neurons, there will be 2^ points in the connection space. 
The synaptic connection matrix is a mapping fi-om the connection space to real numbers. For 
example, the action of a neuron can be: 
Xv' = f (1. aw^ aikr Xi . aiikXiJCKaiiikXiJCiJCi.^^ 

Where Xj^ are neuron states (groimded or excited) at a time step, XN are neuron states at the next 
time step, ak .aik. aiik.aiji i^ ... are connection matrix elements, and f ( xl. x2. x3. are a function 
which governs the neuron transition. The points in the connection space is represented as 
follows: assume there are four binary neurons ( 0. 1. 2. 3) and the set of all possible connections 
is ( nothing: 0. L 2. 3: 01. 02. 03. 12. 13. 23: 012. 013. 023- 123: 0123). where (0. 1. 2. 31 are 
self connections, (01. 02. 03. 12. 13. 23) are regular connections, and (012. 013. 023. 123: 
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01231 are the high order connections. These connections are: 0000 = nothing, 1000 = neuron O's 

self-connection, 0100 = neuron 1 's self-connection 1 100 = connection between neuron 0 and 

1 i.e. the connections are (0000, 1000. 0100, 0010, 0001, 1100. 1010. lOOL QUO. OlOL 

0011, 1110, 1101, 1011, 0111, 11111. 

An input vector for a neural network or an input vector is a vector built from an image pixel 
array. The number of pixels, in general, does not agree with the number of neurons. Mapping an 
input image into a given neural net will be necessarv. For example, let an 8-pixel grey image 
have the pixel array riO, 20, 30, 40, 50, 60, 70, 80) and let a 4-neuron net be (0, 1, 2, 3), the 
input vector can be obtained by pixel average: (15, 35, 55, 75). 
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SUMMARY OF THE INVENTION 

The present invention is different from Informix database where images can be queried by aspect 
ratio, brightness, global colour, local colour, shape, and texture attributes. The present invention 
is different from hnatch where images can be queried by colour similarity, colour and shape 
(Quick) , colour and shap e (Fuzzy) , colour percentage, and colour distribution. The present 
invention is different from the OracleS/ Enterprise Server where images can be queried by color, 
local color, shape, and texture attributes. 

The present invention is unique in its sample image, control process, control parameters, and 
algorithms. The current algorithms do not use methodologies deployed in the above systems. In 
particular, the following parameters are not used: aspect ratio, brightness, global colour, local 
colour, shape, colour similarity, colour and shape (Quick) , colour and shape (Fuzzy) , colour 
percentage, and colour distribution, local color, shape, and texture attributes. The present 
invention has nothing in common with any existing system. 

Even though the current invention is appUed to images, the algorithms in the invention can be 
apphed to other types of data, such as sound, movie, . . . 

1. Process 

The present invention is a content-based image retrieval/recognition system, where users specify 
an image(s) or segment(s); adjust control parameters of the system, and query for all matching 
images from an image directory or database. The user operation is generally divided into two 
phases: learning phase and search/recognition phase. In the learning phase, various types of 
processes, such as image preprocessing, image size reduction, and image filtering are applied to 
the images. Then the images are sent to a recognition module to teach the module the 
characteristics of the image as specified by an array of pixels. Each pixel is defined by an 
integer, which can have any nxmiber of bits. The learning module can use ABM or APN leaming 
algorithms to learn the sample image. Both the algorithms will be listed in the present invention. 
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In the search/retrieval phase, the recognition module decides the classification of an image in a 
search directory or a search database. 

In a retrieval/recognition system, a "training" for the system or "learning" by the system is to 
teach the system what characteristics of an image, or a segment of an image (key) to look for. A 
system operator completes this step by specifying the sample image(s); specifying the parameters 
and clicking one button, the "training" button, which appears in the graphical user interface of 
the system. A "retraining" by the system is to teach the system what characteristics of images to 
look for, after the system is akeady trained. Training and retraining together allows the system to 
leam from many sample image(s) and segment(s) simultaneously. 

A "search" or "retrieval" is to look for matching images from an image source such as, directory, 
many directories, subdirectories, network, Internet, or database, etc. A system operator 
completes this step by specifying the image source such as search directory(s), specifying the 
parameters and clicking one button, the "searching" button, which appears in the graphical user 
interface of the system. The results can be displayed within the software systems or displayed in 
a program created by the system. Two particular applications ar e imag e vorifioation (1:1 
matching, binar>^ output: y e s/no) and image id e ntification (1:N matching, singl e output to 
indicat e a claGsificationV Several applications are Usted below: 

Verification 

Verification is a one-to-one (1:1) Matching of a single sample image set against another. 
Generally, the first sample is newly captured and the second is the enrolled identifier on 
file for a particular subject. For example, in a user authentication environment, a score 
exceeding the threshold would return a 'match', resulting in the authentication of the user. 
A score below the threshold would return a 'no-match\ resulting in the denial of access. 

Identification 

Identification is a one-to-manv (1:N) Matching of a single sample image set against a 
database of samples, with no declared identity required. The single image is generally the 
newly captured sample and the database contains all previously enrolled samples. Scores 
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are generated for each comparison, and an algorithm is used to determine the matching 
record, if any. Generally, the highest score exceeding the threshold results in 
Identification. 

Search 

Search is similar to Identification, i.e. one-to-many (1:N) Matching: however, the result is 
a set of possible matching images, not a classification. Identification returns a 
classification, while Search returns multiple matched images. 

Retrieval 

Same as Search. 

Classification 

In the above three cases: only one class of image(s) is compared with a set of images to 
be searched. "Classification" is many-to-one fNil) or manv-to-many fN:N) Matching. It 
specifies one image in one of many categories. 

A "classification" or ^'recognition" is to repeat training and search for each category of images. 
At the end, a system operator clicks one button, the "classification" button, which appears in the 
graphical user interface of the system. The results can be displayed within the software systems 
or displayed in a program created by the system. Classification is an N: N matching with a single 
output to indicate a classification. 

The parameters and settings of a particular operation can be saved and recalled later. Clicking a 
button, cut and paste, open files, or typing can be achieved by recalling can achi e v e r e calling a 
saved operation. The saved results are called "batch code". The "Batch" buttons provide means 
to execute these saved batch codes. 

A "process" is a sequence of training and searching, or a classification, or a specification of a 
batch code and execution of a batch code. They are fiirther divided into a search process, a 
classification process, and a batch process. 
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After the operator completes a process, the results consists of a list of pairs; the pairs consist of 
the matched image and the 'Veight", which reflects how closely the selected image matches the 
sample image(s). This list can be sorted or unsorted. This list provides the link to the matched 
images so the match images can be viewed with a single click. 

"System integration" is to combine a software component, which is an implementation of this 
invention, with an application interface. 

The search process, which is appUcable to retrieval, verification, and identification, is: 

1 . Enter key image into the system; 

2. Set training parameters and click the training button to teach the system what to look for; 

3. Enter search-directory(s); 

4. Set search parameter(s), and click the search button; 

5. The system output is a list of names and weights: 

• The weight of an image is related to the characteristics you are looking for (the weight is 
similar to an Intemet search engine weight); 

• Click the name of each image and an image will pop up on the screen. 

Figure 1 is the flow chart version of this algorithm. 

The classification process is: 

1 . Enter key image into the system; 

2. Set training parameters and click the training button to teach the system what to look for; 

3. Enter search-directory(s); 

4. Set search parameter(s), and click the search button; 

5. Repeat the above process for each class and then click the "Record" button. At the end, click 
the "Classification" button. The output web page will first list the sample images for each 
class. Then it will list: 

• An image link for each image in the search directory; 
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• The classification weights of this image in each search; and 

• The classification of this image as a link. 

The classification process consists of several Search processes. Each class has a kev image. This 
key image is used to match against all images in a search-directory. The results consist of a set of 
matched images and their matching scores. An image in the search-directory can be matched 
with several kev images. The highest weight determines the final classification of an image in the 
search-directory. Figure 7 is the flow chart version of this algorithm. 

The batch code for a search process is a text file that consists of: 

1. Key image to train the system; 

2. Training parameters: 

3. Search-directory to specify a set of images to be searched: 

4. Search parameters. 

The batch code for a classification process is a text file, that consists of several sets of 
parameters, one set for each class. Each set of parameters consist of: 

1. Key image to train the system: 

2. Training parameters: 

3. Search-directory to specify a set images to be searched: 

4. Search parameters. 

The batch code is generated automatically: cUck the Save button to save the current setting, 
including key(s), search directory(s), and parameters, into a batch code. The batch code can be 
viewed bv chcking a file button to recall one of the many batch codes saved earlier. The batch 
execution can duphcate the Search Process and the Classification Process in two clicks in a 
software implementation: the first click opens the proper batch code that needs to be executed 
and the second click runs the batch code. 

The batch process is: 

1. Provide the batch code to the system, which includes: 
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• Click the save button to save the current setting, including key(s), search directory(s), and 
parameters into a batch code. 

• Click a file button to recall one of the many batch codes saved earher. 

• Cut and paste or simply type in a batch code by keyboard. 
2. Click batch button to execute the code. 

Figure 8 is the flow chart of the batch process. 

An integration process is to combine a software component, which is an implementation of this 
invention, with an application interface. This invention also specifies a user-graphical-interface 
for the integration. 

2. Parameters 

The search, classification, and batch processes require a set of parameters. All the parameters can 
be specified in the system user interface, either through clicking buttons or through Windows. 
The parameters are specially related to the ABM and APN algorithms, which will be claimed in 
this patent. Figure 3 and 4 show a sample implementation of these parameters. 

The "Area of Interest" specifies an image segment, which is specified by 4 numbers: the coordinates of 
the upper-left comer and the bottom-right comer. 

The "intemal representation" specifies the dimensions of a pixel array used for computation, 
which may or may not be the actual image pixel array. 

The "Background" or "Background filter" selects an image-processing filter the pixel array must 
pass through before entering the learning component of the system. 
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The "Symmetry" represents similarity imder certain types of changes, such as intensity, 
translation symmetry. Scaling, Rotation, oblique, combined rotation and scaling or any 
combination thereof. 

The "Rotation Types" specify the range of rotation if the rotation symmetry is used. Examples 
are 360°-rotations, -S"" to 5*^ rotations, and -10° to 10*" rotations, or other settings that fit the 
user's need. 

The "Reduction Type" specifies the method used when reducing a large image pixel array to a 
smaller pixel array. 

The "Sensitivity" deals with the sample segment size; high sensitivity is for small segment(s) and 
low sensitivity is for large segment(s). 

The "Blurring" measures the distortion due to data compression, translation, rotation, scaling, 
intensity change, and image format conversion. 

The "Shape Cut" is to eliminate many images that have different shapes fi-om the sample 
segment. 

The "Extemal Weight Cut" is to list only those retrieved images with weights greater than a 
certain value. The weight Cut is an integer greater than or equal to 0. There is no limit how large 
this integer can be. The "Intemal Weight" Cut plays a similar role as the Extemal Cut in a 
percent value rather than an absolute weight value. 

The "Image Type" specifies the learning component whether to treat the pixel array as black and 
white images or a color image. It also instructs the leaming component whether to use a 
maximum value, integration, or both. 

The "L/S Segment" (Large/Small segment) specifies the system where to focus when searching 
images. 
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The "Short/Long" search specifies an image source such as whether to search one directory or 
many directories. 

The "Short Cut" is a Scrollbar to select an integer between 0 and 99; each integer is mapped to a 
set of predefined settings for the parameters. 

The "Border Cut" controls the portions of images to be used in the image recognition. 

The "Segment Cut" controls the threshold used to reduce an image into an intemal 
representation. 

3. System Layout 

Attrasoft Component-Object structure consists of three layers (See Figure 2): 

• Application Layer 

• Presentation Layer 

• ABM Network Layer 

The ABM Network Layer has two algorithms to be claimed in the present invention: 

• ABM (Attrasoft Boltzmann Machine); 

• Attrasoft PolyNet (APN): multi-valued ABM. 

This layer is responsible for learning and classification. 

The Presentation Layer is an interface between the ABM net layer and the user interface layer. 
There are two types of data used by the systems: user data or appUcation data, and ABM neural 
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data. ABM networks use ABM neural data. User data depends on the application. The 
presentation layer converts the image data into neural data used by the ABM layer component. 

The Application Layer is the front-end graphical user interface, which the users see directly. This 
layer collects all parameters required for necessary computation. 

4. Algorithms 

The ABM layer deploys two algorithms, ABM and APN. The ABM and APN algorithms consist 
of a combination of Markov Chain Theory and the Neural Network theory. Both theories are 
well known. The ABM and APN algorithms are newly invented algorithms, which have never 
been published. 

The following terms are well known: Markov chain, state of Markov chain, invariant 
distribution. 

The basic flow chart for ABM and APN algorithms are: 

1 . Combine an image and its classification into a vector. 

2. All such together form a mathematical configuration space. Each point in such a space is 
called a state. 

3. A Markov chain exists in such a space where the state of the configuration space is a state 
of the Markov chain. 

4. The constmction of such a Markov chain is by a particular type of neural network, called 
ABM network or APN network. This type of neural net satisfies 3 features: (1) fiiUy 
connected; (2) the order of the neural net is the same as the number of neurons in the 
network, i.e. the number of connections is an exponential fimction of the number of 
neurons; and (3) the connections follow particular algorithms, known as ABM and APN 
algorithms. 
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5. The Markov chain will settle on its invariant distribution. A distribution function is 
deployed to describe such a distribution. In particular, such distribution function 
classifies the images. 



^. Th e Markov chain will s e ttl e on its invariant distribution, A distribution function io 
d e ploy e d to describ e such a distribution. In particular, such distribution function 
classifies tho images. 

5. The construction of such a Markov chain is by a particular typo of n e ural network, call e d 
ABM n e t^^ork or APN n e twork. This typ e of n e ural n e t satisfies 3 f e atures: (1) fully 
conn e ct e d; (2) the ord e r of th e n e ural n e t is th e sam e as the numb e r of n e urons in th e 
n e twork, i. e . th e numb e r of conn e ctions is an e xpon e ntial function of th e number of 
neurons; and (3) th e ooim e ctions follow particular algorithms, Icnown as ABM and APN 
algorithms . 

Explanation: 

In Step 1, the explanation on how to combine an image and its classification into a vector 

is given in the TECHNICAL BACKGROUND section. 

In Step 2, no particular action will need to be taken. This is a conceptual step, which 
views an image as a point in a space or several spaces, resulting in the definition of 
distances between points. The distance is further used in various parameters and 
matching computations. The explanation on how to treat an image as a point in an image 
space is given in the TECHNICAL BACKGROUND section. 

In Step 3, a Markov chain is formed. Again, no particular action will need to be taken. 
This is a conceptual step, which specifies how the image matching will be implemented 
via the invariant distribution of a Markov chain. 

In Step 4, an artificial Neural Network is formed to implement the Markov chain. Both 
the ABM and the APN satisfies 3 features: 

(1) FuUv coimected: 

(2) The order of the neural net is the same as the number of neurons in the network. 

i.e. the number of connections is an exponential function of the number of 
neurons; and 
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(3) The connections follow particular algorithms, known as the ABM and the APN 
algorithms> 

Figure 10 gives an example of a fully connected artificial neural network with 4 neurons. 
Figure 1 1 gives an example of a Markov chain generated bv a neural net with 4 neurons 
in Figure 10. The controlling factor in a Markov chain is the transition-probability 
Matrix. Assume the neurons are (3. 2. L 01 in Figure 10. Each neural state is a state in 
the Markov chain. These states are: 0000 = 0: 0001 = 1: 0010 = 2: 1111 = 15. The 
states are 0000 = 0: 0001 = 1: 0010 = 2: 1111 = 15, i.e. the Markov chain state 0 is a 
neuron state where all neurons are grounded: the Markov chain state 1 is a neuron state 
where all neurons are grounded except neuron 0: the Markov chain state 2 is a neuron 
state where all neurons are grounded except neuron 1: the Markov chain state 3 is a 
neuron state where neurons 2 and 3 are grounded and neurons 0 and 1 are excited: . . . 
In Step 5. the Markov chain will settle on its invariant distribution, which can be used for 
identification. Let x be an image, and assume there are two classes: class 0 stands for No 
Match and class 1 stands for Match: then the two possible vectors are (x. 0) and (x. IV 
Let an invariant distribution function of the Markov chain, constructed bv the neural 
network, be z = F (v). where v is a vector. If v = (x. 0). z = zO: and v = (x. 1), z = zl. then 
the probability of x in class 0 is zO and the probability of x in class 1 is zl. The result will 
be Kx. 0. zO). (x. L zD). The (x. K zU will be the computation result. For simplicity, (x, 
L zl) is further simplified to (x. zl). The users will see results (x. zl) directly in the 
out put of the system. Figure 5 shows a possible implementation of the Search process 
results and Figure 6 shows a possible implementation of the Classification process 
results. 



23 



Th e St e p 4 of th e abov e is d e fin e d as follows: 

L e t X b e an imag e , and lot a, b b e two class e s; th e n th e two possibl e v e ctors are (x, a) and (x, b). 
Let a distribution function b e z ~ F (y), wh e r e y is a v e ctor. If y - (x, a), z ~ zl; and y - (x, b), z 
~ z2, th e n th e probability of x in class a is zl and th e probability of x in clasG b is z2. Th e r e sult 
will be ((x, a, zl), (x, b, 22)] . The us e rs will s ee r e sults lik e this dir e ctly in the output of the 
syst e m. 

In the ABM or APN algorithms, content-based image retrieval and image recognition are 
basically the same problem; therefore, they can be converted from one to the other. To convert 
from an image search problem to an image recognition problem, one query is required for each class. To 
see whether an image, say B, is in class A, you first train ABM with all images in class A, then try to 
retrieve image B. If image B is not retrieved, then image B is not in class A. If image B is retrieved only 
for class A, then image B is in class A. If image B is retrieved for several classes, the class with the 
largest relative probability is the one to which image B belongs. Image search is an image classification 
problem with only 1 class. 

ABM is a binary network. APN is a multi-valued network. 

5. Components and Application-Programming Interface 

Software components can be isolated to be attached to different front-end systems. Figure 2 
shows the three-laver architecture of this invention. This can be done with ABM neural layer 
alone, or both ABM layer and presentation layer. The ABM layer component is a core of the 
present invention. The value of such a sub-system is the same as the whole system. 

This invention also defines the application-programming interface (API), which specifies the 
system integration. This API is called IVI-APL 
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BRIEF DESCRIPTION OF VIEWS OF THE DRAWING 

Figure 1 shows the algorithm of the Search Process, which is applicable for image verification, 

identification, and retrieval. 

Figure 2 shows a 3-Layer Internal Architecture. 

Figure 3 shows a sample User Interface of the Present Invention. 

Figure 4 shows a sample Key Input for the Present Invention. 

Figure 5 shows a sample Search Output of the Present Invention. The search output is a hst of 
pairs. 

Figure 6 shows a sample Classification output of the Present Invention. The classification output 
is a Ust of triplets. 

Figure 7 shows the Classification Process, which consists of a multiple-search process in Figure 
L 

Figure 8 shows the Batch Process, which allows users to duplicate a Search or Classification in 
two clicks. 

Figure 9 shows the ABM and APN Algorithm Flow Chart. 

Figure 10 shows an example of a fully connected artificial neural network with 4 neurons. 

Figure 1 1 shows the Markov chain generated bv the neural net with 4 neurons in Figure 10. The 

controlling factor in a Markov chain is the transition-probabilitv Matrix. Each neural state is a 

state in the Markov chain. The states are 0000 = 0: 0001 = 1 : 0010 = 2: . . . : 1 1 1 1 = 15. 

Figure 12 shows the More Detailed ABM Algorithm Flow Chart. 

Figure 13 shows the More Detailed APN Algorithm Flow Chart. 

Figure 14 shows the Connection Space and the Sensitivity distance. 

Figure 15 shows the hnage Space and the Blurring distance. 

Figure 16 shows the ABM and APN Leaming Algorithm Flow Chart. 
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Figure 17 shows the ABM Recognition Algorithm Flow Chart. 
Figure 18 shows the APN Recognition Algorithm Flow Chart. 



26 



DETAILED DESCRIPTION OF THE DISCLOSED EMBODYMENT 



Preferred Embodiment of the Search System 

An image search/classification constructed in accordance with the preferred embodiment 
comprises a computer-based workstation including monitor, keyboard and mouse, a content- 
based image retrieval software system and a source of images. 

The source of the images may be on the local drive, network or the Internet. The source is 
connected to the workstation. The source of images may be accessed directly via open files, or 
indirectly, such as going into a file to find the images or going into a database application to find 
the images, etc. 

The preferred workstation can be a PC or any other type of computer eefflp«teFS, which connects 
to a data source. 

The preferred content-based image retrieval software system is any software, which has ABM or 
APN algorithm as a component. It can be a Window-based system, or any other operating system 
based systems, or hitemet based systems. 



Overview of the ABM Algorithm 

The following terms are well known: synaptic connection or connection. 
The basic flow chart for ABM algoritimi is: 

1 . Create an ABM net with no connections; 

2. Combine an image and its classification into an input vector. 

3. Impose the input vector to the learning module. 
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4. The ABM neural connections are calculated based on the input vector. Let N be the 
number of neurons; the order of connections can be up to N and the number of 
connections can be 2**N, where ** represents the exponential function. 

5. The Markov chain is formed after the connections are established. This Markov chain 
will settle on its invariant distribution. A distribution function is deployed to describe 
such a distribution. 

6. This distribution fiinction, once obtained, can be used to classify images. This will 
produce triplets of image, class, and weight. Image retrieval and classification are two 
different sides of tiie same token. 

7. These triplets of image, class, and weight can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 

one class. 
Explanation: 

1 An empty neural net is created - Tt looks Uke the eraoh in Figure 10 without any 
connections. 

Ste p 2. The explanation on how to combine a n image and its classification into a vector is 
piven in the TECHNICAL BACKG ROUND section. 

Step 3. The ABM net will read an input vector. The P resentation Layer in Figure 2 
pr ^;.rfis the input vector. In general, th e image size and the ABM neural net size do not 
matr.li' fnr example, an image is 480x6 40 and a neural net is 100x100. The Presentation 
Ts^ yetr in Figure 2 will reduce the image pixel array to fit the ABM neural network 
thmn ph yarious methods. This procedure will be specified in more detail later in the 
"Presentation l a yer . . ." section. 

Ste p 4. The ABM will be trained. This step will be fu rther expanded later in the "ABM 
Training Algorithm" section. 

Ste p S. The ABM net generates a Markov chain. The ABM net is a fiilly connected neural 
net following the rule of the Boltzmann M achine, i.e. each neuron has a certain 
pmhahilitv to be excited and a certain probabilit y to be grounded. Such a neural net is a 
Markoy chain. Figures 10 and 11 show the r elationship between a neural net and a 
Markoy chain. No particular action will need to be taken in this step. This is a conceptual 
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step, which specifies how the image matching will be implemented via the invariant 
distribution of a Markov chain. 

Step 6. An invariant distribution of the Markov chain is reached, which can be used for 
identification. The explanation on how to classify images via the invariant distribution is 
in the TECHNICAL BACKGROUND section. 
Step 7. The results are presented. 
Figure 12 shows the ABM Algorithm Flow Chart. ABM is for black-and-white images and APN 

is for color images. The basic approach, including the base Markov chain, is identical. The only 

difference is that the APN will construct two extra vectors, one firom the training image and one 

from each search image. These two vectors will be used to modify the scores of the ABM 

algorithm, i.e. the invariant distribution of the Markov chain. 

Overview of the APN Algorithm 

The APN algorithm is an extension of the ABM algorithm. The ABM algorithm is for binary images and 
the APN algorithm is for multi-valued images. The APN algorithm will first use the ABM algorithm to 
make an image matching for binary images. If there is no match from the ABM algorithm, the ABM 
algorithm will output a score 0, which will remain to be 0 when the APN algorithm finishes. If there is a 
match for the ABM algorithm, then the APN will further modify the ABM results by comparing two 
vectors generated by comparing two images by the two comparing im ages. For example, consider two 
images (3 5 0 0) and (1 2 0 0\ their binary version are 1 100 and 1 100. The ABM algorithm will compare 
the binary image. 1100 OftH- and 1100. and return a match. The APN algorithm will take the ABM 
results, and modify it bv comparing (3 5) and (1 2). 

The basic flow chart for APN algorithm is: 

1 . Create an APN neural net with no connections; 

2. Combine an image and its classification into an input vector. 
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3. Impose the input vector to the learning module. 

4. The APN neural connections are calculated based on the input vector. Let N be the 
number of neurons; the order of connections can be up to N and the number of 
connections can be 2**N, where ** represents the exponential function. 

5. A mapping over each connection is established. Let K be a number of neurons in a K 
order connection, where K is less than or equal to N, then there this will be a K to K 
mapping, i.e. the domain of the mapping has K integers and the range of the mapping has 
K integers. 

6. The K-elements mapping is changed to N-element mapping by adding (N - K) pairs of 0 
to 0 relations for each of the neurons not in the set K. By taking the domain of this 
mapping away, the range of this mapping forms a vector, APN connection vector. 

7. The Markov chain is formed after the connections are established. This chain will settle 
on its on its invariant distribution. A distribution function is deployed to describe such a 
distribution. 

8. This distribution function, once obtained, can be used to classify images. This will 
produce triplets of image, class, and weight. 

9. Comparing the input-vector and the APN-connection-vector modifies this weight. This 
will produce a new set of triplets of image, classification, and weight. 

10. These triplets of image, class, and weight can be viewed as the results of the 

classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 
Explanation: 

Step L see Step 1 of the ABM net above. 

Step 2- see Step 2 of the ABM net above. 

Step 3, see Step 3 of the ABM net above. 

In Step 4, the underlvine ABM will be trained. See Step 4 of the ABM net above. 
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Step 5 represents the extensions from a binary ABM net to a multi-value APN net. In the 
TECHNICAL BACKGROUND section, we have stated the black-and-white image is a 
subset and the color image is a subset of particles with mass. The K-to- K mapping in this 
step represents the mapping from a particle to its mass; therefore, the images us ed are not 
black-and-white images, but color images. For a given training im age, this mapping is 
defined bv the input image. For example, let N = 4 and K = 3, and let a training i mage be (10. 
20. 30, 0\ then the base black-and-white image is 1 110 and the mapping is fO ^ 10. 1 ■»20, 
and 2-»30i. For both the training image and search image, this K-to-K map ping will be 
computed in a similar way. 

In Step 6, the K-to-K mapping will be modified twice. It will first be converted into N-to-N 

ma pping first. In the above example, this new mapping is (0 -» 10, 1 ^20. 2-»30, 3 ^ 01. 

Then it will be normalized. For example, the new mapping is the old mapping divided bv 2: 

(0 -> 5. 1 ->10, 2^15, 3 -» 01. Now the Domain of the mapping, (0, L 2, 31 is removed 

and the Range of the mapping converted into a vector 15, 10, 15, 0). This APN connection 

vector is the difference between the ABM net and the APN net. 

Step 7, see Step 5 of the ABM net above. 
Step 8. see Step 6 of the ABM net above. 

Step 9. the results in Step 8 are modified in this step. The distance (Ln -distances, where p = 

L 2 oo) between the new APN connection vector generated bv an input search image and 

the APN connection vector generated bv the training vector is computed. This distance 
modifies the resuUs in Step 8. It could be as simple as dividing the old weights bv the 
distance. 

Step 10. see Step 7 of the ABM net above. 
Figure 13 shows the ABM Algorithm Flow Chart. 
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User Interface Layer of software for implementation of ABM and APN 
Algorithms 

There are three major operations: 

• Search or retrieval; 

• Classification; and 

• Batch. 

These are the principle modes of the system that runs on the workstation. The software executed 
in these three modes can have various user interfaces, such as in Windows environment or the 
web environment, etc. The user interface collects necessary information for the computation. 

A Search process has two phases: learning and recognition, hi the leaming phase, a key image is 
used to train the proposed system what to look for by using the ABM or th e APN leaming 
algorithm. In the recognition phase, the proposed system searches through all im ages in a search- 
directory or a search-database for comparisons via the ABM or the APN recog nition algorithm. 
The sample outputs are given in Figure 5 and Figure 6.. The User Inte rface Layer is responsible 
for collecting all parameters, including key images and sources of images. The User Interface 
Layer will also pass the parameters to the translation layer, which convert s an image into an 
input vector. The following are the parameters: 

Oth e r than tho key and th e a sourc e of images, th e uGor int e rface may or may not pass th e 
following information to the next layer: 

The "Area of Interest" specifies an image segment by two clicks. These two clicks generate 4 numbers 
the coordinates of the upper-left comer and the bottom-right comer. See a sample implementation in 
Figure 4 for the coordinates of the upper-left comer and the bottom-right. 

The "intemal representation" specifies the dimensions of a pixel array used for computation, 
which may or may not be the actual image pixel array. 
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The "Background'' or "Background filter" selects an image-processing filter the pixel array must 
pass through before entering the learning component of the system. The interface will be 
responsible for selecting one of many available filters. 

The "Symmetr/' represents similarity under certain types of changes, such as intensity, 
translation symmetry, Scaling, Rotation, oblique, combined rotation and scaling or any 
combination thereof For the translation symmetry, this is implemented by physically translating 
the sample image to all possible positions. The similar methods can be apphed to other 
symmetries. 

The "Rotation Types" specify the range of rotation if the rotation symmetry is used. Examples 
are 360°-rotations, -5° to 5° rotations, and -lO"^ to 10° rotations, or other settings that fit the 
user's need. 

The "Reduction Type" specifies the method used when reducing a large image pixel array to a 
smaller pixel array. 

The next few variables are Sensitivity. Blurring, and Shape Cut. These are various distances 
calculated bv the system in the image space or the connection space, which are introduced in the 
TECHNICAL BACKGROUND section. The users set up the thresholds for these distances and 
the system computes these distances between two images. If a distance is gr eater than the user- 
specified value, the images do not match. Within the user specified range> the smaller these 
distances are, the higher the matching scores will be. 

The "Sensitivity" deals with the sample segment size: high sensitivity is for sma ll segment(s) and 
low sensitivity is for large segment(sV This is a method to limit the relevant ne ural connections. 
The connection space is given in the TECHNICAL BACKGROUND section. When the ABM 
net. xL is trained, there will be certain coimections. All possible connections together form a 
space, HI. For the ABM net with N neurons, such a space will have a maximum of 2**N 
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connections, where ** is the exponential function. Each tra ined ABM net will have a set hi. 
representing non-zero connections. When deciding whether a n image. 12. in a search directory is 
a match to the current sample image. II. this image 12 can be turned around to train the new but 
similar ABM neural net. x2. This will generate a set of connect ions. h2. Sensitivity determines a 
maximum distance, d. between hi and h2. Given a search image, whi ch generates h2. if the 
distance between hi and h2 is greater than the Sensitivity, then there is no match between the 
input search image and the kev image. Users set Sensitivity as a para meter of the proposed 
system: see the Sensitivity button in a sample implementation in Figure 3. 
For example, assume there are four binary neurons I 0. 1. 2. 3) and tiie set of all possible 
connections is I notiiing: 0. 1. 2. 3: 01. 02. 03. 12. 13. 23: 012. 013. 023. 1 23: 0123k where (0. 
1. 2. 31 are self connections. 101. 02. 03. 12. 13. 23\ are regular connectio ns, and 1012. 013. 
023. 123:0123) are tiie high order connections. These numbers are the subscripts of a synaptic 
connection matrix element that define the action of a neuron: 
Xi.' = f (1. ai,,aiirXi ^ai|kXi Xi.ajjikXi XjJCi.^^ 

Each connection in this space is considered as a point. A ssnnne one image generates a set, hi = 
(01. 02. 03. 12. 13. 23k and a second image generates the same set. h2 = 101. 02. 03. 12. 13. 
23). Assume the Li distance is used for Sensitivity distance, where L i = I x? - xi I + I v? - vi I 
+ ■ . .. tiien tiie Sensitivity distance between hi and h2 is 0. Assume one image genera tes a set, hi 
= (Oil and a second image generates a set. h2 = (0123k tiien tiie Sensitivity distance between 
hi and h2 is 2. because the distance between 1100 and 1111 is 2. The best way to compute tiie 
Sensitivity distance is to use flie connection space in tiie TECHMCAL BACKGROUND 
section, which converts the connections into binary strings. For e xample. 01 ^ 1100. 02 ^ 
1010. 03 1001. Figure 14 shows the connection space. 
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Tho "SonGitivit/' doalo with tho oamplo oogmont oizo; high oonoitivit>^ io for Gmall oogmont(o) and 
low Gonoitivity is for largo oogmont(o). Thio io a method to limit tho relevant noural connoctiono. 
When ABM not, xl, io trained, thoro will bo oertain oonnootiono. All poGaiblo oonnoctions 
togothor form a opaco, HI . For th e ABM not with N nourono, ouoh a opaco will have a maximum 
of 2'*^'>'N point, whoro "^"^ io tho exponontial function. Each trained i\BM not will have a oet hi, 
roprooonting non zoro connoctiono, ^\^lon d e ciding whothor an imago, 12, in a ooarch diroctor>^ is 
a match to tho ourront oamplo imago, II, thio image 12 can bo turned around to train tho now but 
similar ABM neural not, x2. The will generate a oet of oonnootiono, h2. Similarit>^ determineo a 
maximum diotanoe, d, either using the Hauodorff diotanoo or LI diotano e or L2 distanc e . In the 
oonneotion opaco, starting from tho oonnection set, h2, of tho new ABM net, after appl>ing thio 
now distanco, d, a now sot, h3, is obtained. Obviously tho smaller thio distance, d, io, the omall e r 
this now o e t, h3, will bo. Thio now oot, h3, io thon tranoformod back to hi. Any point in hi but 
not in h3 will bo conoidorod "too far" and thoroforo io sot to Q for tho current imago, 12, in th e 
search dir e ctory. Thio reduction in the oonnectiono opaco is determined by the oonoitivity. 

The "Blurring" measures the distortion due to data compression, translation, r otation, scaling, 
intensity change, or image format conversion. For example, when an image in the bitmap format 
is saved into the ineg format, the pixel values changes sUghtlv. which are not visible to human 
eves. An image is a point in an image space. The image space is given in the TECHNICAL 
BACKGROUND section. All possible images (points^ together form a space, the image space. 
An image is a point in such a space. When deciding whether an imag e. 12. in a search director/ is 
a match to the training image. 11. the distance, d ai. 12^ can be c alculated. Blurring enlarges this 
comparison. In the image space, anv unage in I2's neighborhood is iu st as good as 12. Blurring 
determines the radius of this neighborhood. I2's neighborhood consists of a ll images with a 
certain distance from 12. The distance can be either the Hausdorff distance, or LI distan ce, or L2 
distance (See the TECHNICAL BACKGROUND section! or multiple dist ances. The maximum 
radius to define the neighborhood is Blurring or Blurring distance, which is spe cified bv the user 
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as a parameter, i.e. users enter a value for this parameter in th e user interface: see the Bliirring 
button in a sample implementation in Figure 3. For exa mple, let x =1 1 1 1 be a four-pixel binary 
image, the following images have a distance of 1 from the o riginal image x: 0111, 1011, 1101, 
1 1 10. Figure 15 shows the binary image space. A "sphere" with r adius 1 around x will consist of 
acenterx=llll.and0111. 1011. 1101.1110. 

Tho "Blurring" moaouroo the diotortion duo to data oomproooion, tranolation, rotation, ocaling, 
intensity chongo. and imago format convoroion. Thio method oxpando an imago in tho ooaroh 
diroctory from a single point to a oot oo foUowo. All poooiblo imagoG together form a opaco, th e 
imago spaoo. An imago is a point in ouch a opoo e . Whon d e ciding wh e ther an imag e , 12, in a 
ooaroh dir e ctory io a match to tho ourront oamplo imago. II, thio imago 12 can bo tumod a omall 
oot around tho 12. Lot tho oot bo IS2. Blurring dotorminoo a majcimum diotanoo, d, oithor using th e 
Hauadorff diotanco or LI diptanco or L2 diotanoo. In tho imag e opaco, starting from tho 12, after 
applying thio now diotanoo, d, a now ophor e oot, IS2, io obtainod. Obviously tho omallor thio 
diotanoo, d, io, tho omallor thio now oot, IS2, will bo. Now any point in thio sot, IS2, io just as 
good as 12. This e xpanoion in tho imag e opao e io d e tormin e d by tho Blurring. 

The "Shape Cut" is to elimmate manv images that have different shapes from the sample 
segment. All possible images together form a space, the image space. The image space is given 
in the TECHNICAL BACKGROUND section. All possible images togeth er form a space, the 
image space. An image is a point in such a space. There are several distances defi ned in this 

image space, including Lp -distances, where p = 1. 2 oo and Hau sdorff distance. When 

deciding whether an image. 12. in a search directory is a match to the curren t sample image. II, 
the distance, d (II. U) can be calculated. "Shape Cut" determines a maximum dist ance. D. either 
using the Hausdorff distance. LI distance. L2 distance, or multiple distances . If the distance 
between II and 12 is greater than "Shane Cut", d (II. 12) > D. then there is no match between the 
input search image and the key image. A user specifies the Shape Cut as a par ameter, i.e. users 
enter a value for this parameter in the user interface (Figure 3 and Figure 4). 
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Tho "Shape Cut" io to flimi>"*" "'""y i"^"C"" thnt hnvo difforont ohapoo from tho oompl e 
aogmont. All posoiblo imagoo togothor form a opao e , tho image opaoo. An imago io a point in ouch 
a opaoo. Whon dooiding whothor on imago, 12, in a oooroh dirootory io a match to tho current 
oamplo imago, II, tho diotanoo botwoon II and 12, d, can bo dotorminod, oithor uoing the 
Ilauodorff diotanco or LI diotonco or L2 diotanco. If thio diotanco, d, io larger than a 
prodotorminod diotanoo, D, a miomatoh can bo doolarod v . 'ithout going through tho ABM noural 
not. Thio prodotorminod diotanoo, D, is oot by tho "Shapo Cut" paramotor. 

The "Extemal Weight Cut" is to list only those retrieved images with weights greater than a 
certain value. The weight Cut is an integer greater than or equal to 0. There is no limit as to how 

large this integer can be. 

The "Intemai Weight Cut" plays a similar role as the "Extemal Cut" in a percent value rather 
than an absolute weight value. 

The "Image Type" specifies the ABM or APN algorithm. It also instructs the neural layer 
component how to compute the weights. The weight can be computed by using the invariant 
function of the Markov chain, or integration of all contributions in the time evolution of the 
Markov chain, with or without reaching the invariant distribution. 

The "L/S Segment" (Large/Small segment) specifies the system where to focus when searching 
images. Please refer to the similarity to understand the set of contributing coimections, i.e. not 
every connection is a contributing connection. Small and Large segments deploy different scales 
in the determining the set of connections. 

The "Short/Long" search specifies an image source such as whether to search one directory or 
many directories. 

The "Short Cut" is a Scrollbar to select an integer between 0 and 99; each integer is m^ped to a 
set of predefined settings for the parameters. 
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The "Border Cut" is to eliminate the border sections of unages. This parameter controls the 
percentage of images to be eliminated before entering consideration. 

The "Segment Cuf is best illustrated by examples. Assume a 4- 400x400 image is reduced to 
100x100 intemal representation, as set by the parameter "Internal Representation"; then 16 
original pixels will be reduced into 1 pixel. The new value of the single pixel is determined by 
the parameter "Reduction Type". The "Segment Cut' sets a threshold: if the number of non-zero 
pixels is greater than the threshold, the pixel will have a non-zero value; otherwise, the pixel will 
have a zero value. 

Presentation Layer of software for implementation of ABM and APN 
Algorithms 

Figure 2 shows the position of the presentation layer . The presentation layer transforms the 
image data to neural data. The procedure includes: 

1 . Open files from the image source; 

2. Decode the image into pixels arrays; 

3. Process images with a filter; 

4. Reduce the size of images to an intemal representation. The users can arbitrarily choose 
the intemal representation of the images. Such reduction can be based on individual 
images on a case-by-case reduction, or deploy the same reduction factor across to all 
images. 

5. In the case where many pixels in an image have to be combined into a new pixel before 
leaving this layer, the user can choose a reduction type such as taking average, maximum, 
minimum, or deploy a threshold. The result is an input-vector . 

6. Pass the input-vector imago array to the next lave r. the ABM or APN laver (See Figure 2 
for the three lavers) . 
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ABM Layer of software for implementation of ABM and APN Algorithms 



This Upper level of this layer has two branches: 

• Training Objects 

• High level training class 

• Low level training class and 

• Symmetry class 

• Recognition Objects 

• High level recognition class 

• Low level recognition class 

This lower level of this layer has only one class, the memory management class. 

The purpose of the memory management class is to claim memory space from RAM, 64K at a time. 
This memory space will be used for storing the connections. It also returns the unnecessary space back to 
the operating system of the computer. 

The low level training object is to provide all necessary functions used by the high level training 
class. 

The symmetry object is to implement the symmetry defined earlier. 

The high level training class incorporates symmetry and implements the ABM or APN 
algorithm. The "image Type" parameter in the user interface will determine which algorithm will 
beusedttse. 

The basic idea for the tr ainin g algorithms for both the ABM and the APN is as follows: (1) Break 
a training image into segments; (2) compute the first coimection matrix ele ment from a segment: 
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(3^ Compute the rest of the matrix elements from the first el ement: (4) repeat for all segments. 
Breaking an image into segments can be as simple as dividing an image into 10x10 segments. 
For example, a binary image 11111111 can be dividied into 4 segments : 1100000, 00110000, 
00001 100, and 00000011. 

ABM Training Algorithm is: 

1 . Delete the existing ABM connections; 

2. Combine an image and its classification into an input vector. 

3. The ABM neural connections are calculated based on the input vector. Let N be the N io th e 
number of neurons, these connections can be up to the order of N. The image is randomly 
breaking down into a predefined number of pieces , for example, 10x10 pieces or 8x8 pieces . 

4. Let an image piece, pi, have K = (kl + k2) pixels, where K is an integer. After imposing the 
pixel vector to the ABM net, kl is the number of neurons excited and k2 is the neurons of 
neurons grounded. A neural state vector can be constructed to represent such a configuration, 
which kl components being 1 and k2 components being 0. 

5. All neuron configurations together form a space, the configuration space. All neuron 
connections together form a space, the connection space. Now make the configur ation vector 
= connection vector = pi, which results in the first synaptic connection matrix element. 

6. Calculations of the rest of the matrix elements is based on a distance in the c onfiguration 
space, either the Hausdorff distance or LI distance or L2 distance, and the fi rst matrix 
element calculated in the last step. The defmition of the connection space in the 
TECHNICAL BACKGROUND section yields the definition of a distance between 
connections. Many connection vectors will be in a group with a distance of 1 from pi. Many 
vectors will be in a group with a distance of 2 from pi. A con nection represented by pi is 
assigned the largest value. Those connections in the group of distance 1 will have small 
values: those connections in the group of distance 2 will have even smaller values even 

vdaes After a certain distance, the connection matrix elemen ts will be 0. There are many 

ways to generate these matrix elements from the first matrix element. The present invention 
covers all possible combinations of such a generating method. 
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5. All ouch vootoro togothor form a space, th e connoction opaoo. A distanoo, oith e r the 
HauodorfF diotanco or LI diGtonco or L2 diotanoe can b e defin e d in this opaoo. Such a 
definition of a distance allowo all poosibl e conn e ction v e ctoro to b e olasoifiod via a distanc e 
from pi. Many v e ctors will bo in a group with distanc e 1 from pL Many vootors will b e in a 
group with distance 2 from pi, . . . 

6. The connoction reprooontod by pi is aooignod tho larg e st synaptic connoction weight. Thooo 
oonnoctions in tho distanc e 1 group will hav e small e r weights, .... After a c e rtain distanc e , 
tho connoction woi^to will bo 0, or there will bo no connoctions. Tho prosont invention 
covers all possible combinations of suoh a g e nerating m e thod. 

7. The Markov chain is formed after the connections are established. 
Explanations: 

Stenl. A new neural net is initiated. 

Step 2. The explanation on how to combine an image and its classification into a vector is 
given in the TECHNICAL BACKGROUND section. 

Step 3. The training image is broken into several pieces. For example, an im age I10> 20. 
0. 0, 30. 40. 50. 0. 0. 01 can be broken into two pieces. (10, 20. 0, 0, 301 and 140. 50. 0. 

0. 01 

Step 4. For a given image piece, kl are the excited neurons and k2 are the grounded 
neurons. Assume an image is (10. 20, 0, 0, 30). and it is sent to a neural n et (0. 1. 2. 3, 
4k then kl = 3 and k2 = 2. The excited neurons are 0. 1. and 4. The grounded neurons 
are 2 and 3. The neuron configuration vector is 1 1001. The neural net configuration space 
is explained in the TECHNICAL BACKGROUND section. 11001 is a point in this 
configuration space. 

Step 5. The neural net connection space is explained in the TECHNICAL 
BACKGROUND section. This step makes the configuration vector and the connection 
vector the same, resulting in calculating the first synaptic connection matrix element. In 
the above example, the configuration vector is 1 1001. Assign it to the connection vector. 

1. e. the first non-zero connection is between neurons (0. 1. 41. am 4 = 10 + 20 + 30. Again, 
both the configuration space and the connection space are explained in the TECHNICAL 
BACKGROUND section. 
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Step 6. After the first connection matrix element is calculated, others can be calculated. 
In the above exa mple, the following connection vectors have a distance of 1 from 11001: 
0100L 10001, 11101. IIOIL and 11000. Thev are obtained bv changing one of the five 
bits. The following connection vectors have a distance of 2 fi-om 11001: OOOOK 01101, 

mon. 01000. lOlOL lOOll, lOOOO. inn, moo. IIOIO The computation of new 

matrix elements will be determined bv a function f (a. d\ wh ere a is the first matrix 
element in Step 5 and d is the distance. For example, f (a. d^ = a/(l+dV In the above 
example. aniA = 60 and 01001 has the distance of 1 firom amd ther efore, ai^ = aniA /(l+d)= 
60/f 1 + n = 30. Similarlv. an^ = aid = 30. 

Step 7. Once the connection matrix is calculated, the ABM neural n et is defined. It 
behaves like the Boltzmann Machine, i.e. each neuron has a certain probabilitv to be 
excited, based on the connection matrix. This probabilistic neura l net defines a Markov 
chain. 

The following example shows a comparison among an ABM net, an image, t he input-vector, the 
first connection vector, the first connection-matrix element value, the rest of the connection 
vectors, and the rest of the connection matrix element values: 
Let a 5-neuron ABM net be 10. 1. 2. 3. 41 : 

Let a pixel arrav of an image segment be: 15. 15. 10. 30. 0. 1. 0. 1. 0 601; 
Then an input vector is (10. 20. 0. 0. 301: 
The first connection vector is: 1 1001 : 

The first connection-matrix element value is: ani/i= 10 + 20 + 30: 

The rest of the connection vectors are: 01001. 10001. 1 1 10 1. 1 101 1. and 1 1000: 

The rest of the connection values are: aiA= aflu/(l+d)= 60 /f 1 + 1) = 30. am = au = 30, . . . 

APN Training Algorithm is: 

1 . Delete the existing ABM connections; 

2. Combine an image and its classification into an input vector. 
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3. The ABM neural connections are calculated based on the input vector. Let N be the UAb 
the number of neurons, these connections can be up to the order of N. The image is 
randomly breaking down into a predefined number of pieces. 

4. Let an image piece, pi, have K = (kl + k2) pixels, where K is an integer. After imposing 
the pixel vector to the ABM net, kl is the number of neurons excited and k2 is the 
number n e urons of neurons grounded. A neural state vector can be constructed to 
represent such a configuration, which kl components being 1 and k2 components being 
0. 

5. All neuron configurations together form a space, the configuration space. Al l neuron 
connections together form a space, the connection space. Now make th e configuration 
vector = connection vector = pL which results in the first synaptic connection matrix 
element. 

6. Calculations of the rest of the matrix elements are based on a distance in the 
configuration space, either the Hausdorff distance or LI distance or L2 distance, and the 
fn-st matrix element calculated in the last step. The definition of the connec tion space in 
the TECHNICAL BACKGROUND section yields the definition of a distance between 
connections. Many connection vectors will be in a group with a distance of 1 fi -om pL 
Many vectors will be in a group with a distance of 2 fi-om pi. A connection represented 
bv pi is assigned the largest value. Those connections in the group of distance 1 will 
have small values: those connections in the group of distance 2 will have even smaller 

values oven values After a certain distance, the connection matrix elements will be 0. 

There are many ways to generate these matrix elements from the first matrix element. 
The present invention covers all possible combinations of such a generating method. 

5. All such voctors tog e ther form a space, the connoction space. A digtonco, e ith e r the 
Ilauodorff distanco or LI distance or L2 distance con bo defined in this spaco. Such a 
dofmition of a distance allows all possible connoction v e ctors to bo classifi e d via a 
distance from pi. Many vectors will bo in a group with distanc e 1 from pi. Many voctors 
will bo in a group with distanc e 2 from pi, ... 

6. Th e oonn e otion roprosonted by pi is assigned the largest synaptic oormoction w e ight. 
Those ooimeotiono in the distanoo 1 group will havo small e r w e ights, .... Aft e r a c e rtain 
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distance, th e connection weights will be 0, or th e re will be no oonnootions. Th e present 
invention covers all possibl e combinations of such a g e n e rating m e thod. 

7. The Markov chain is formed after the connections are established. 

8. For each connection, in addition to the synaptic connection weight, a mapping over each 
connection is established. Let kl be a number of neurons in the original kl order 
connection generated by pi, then this mapping maps from the kl neuron to the kl pixel 
value which excited these neurons. This completes the connection for the original 
segment pi. 

9. The segment, pi, also generated many other connections. If a neuron in this connection is 
one of the original kl neurons in pi, then this neuron is mapped into the corresponding 
pixel value, which causes this neuron to be excited; otherwise, this neurons is mapped 
into 0. This completes the mappings of all connections generated by this segment pi. 

The APN Training Algorithm is exactlv the same as the ABM Training Algorithm^ with the one 
exception of storing a mapping associated with each connection. Figure 16 shows the ABM and 
APN Training Algorithm. 

Example 

Let a 5-neuron ABM net be (0, L 2, 3. 4); 

Let a pixel array of an image segment be: 15, 15. 10, 30, 0, 1. 0, L 0 601; 
Then an input vector is (10, 20. 0. 0, 30): 
The first ABM connection vector is: 1 1001: 

The first ABM connection-matrix element value is: aniii= 10 + 20 + 30: 

The rest of the ABM connection vectors are: 01001, 10001. 1 1 101. 1 101 L and 1 1000: 

The rest of the ABM connection values are: au = anM/(l-t-d)= 60/(1 + 1) = 30. 2uu = a^ = 

30..,.: 

The first APN connection is |aoi^= 10 + 20 + 30 = 60. (0-»10, l'»20. 4'»30n. where 
(0^10. l-»20, 4'»30) is the mapping for an^: 

The rest of the APN connections are: (aij^= 30. fl-»20. 4'»30)l, where fl'»20. 4'»30) 
is the mapping associate to ai^(agj^= 30. (0^10. 4'»30n, where (0^10, 4^30) is the 

mapping associate with an^i : . . . 
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The low-level recognition object is to provide all necessary functions used by the high-level 
recognition class. 

The high-level recognition class implements the ABM or APN algorithm. The "image Type" 
parameter in the user interface will determine which algorithm will be used «se. 

The basic idea for the recognition algorithms for both the ABM and the APN is as f ollows: (1) A 
trained ABM or APN net forms a Markov chain, which will settle on an invarian t distribution: 
(2) The distribution function will classify images. 

ABM Recognition Algorithm is: 

1 . An image to be classified is imposed on the Markov Chain. 

2. This Markov , chain will settle on its invariant distribution. A distribution ftinction is 
deployed to describe such a distribution. 

3. This distribution function, once obtained, can be used to classify images. This will 
produce triplets of image, class, and weight. Image retrieval and classification are two 
different sides of the same token. 

4. These triplets of image, classification, and weight can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 

Explanation 

Step 1. An ABM net is alreadv trained with a training image. II, at this po int. The ABM 
net also behaves like a Markov chain. A new image is entered to the ABM net, which has 
an input vector, x. 

Step 2. This Markov chain has an invariant distribution, described bv a function z = f(\\ 
where v looks like this (x, 0) or (x. 1). Here x is an input vector. The distribution function 
will produce triplets (x. 0. zO) and (x. L zl\ where zO and zl mav or mav not be 0. 
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Step 3. The invariant distribution function actually classifies the image as follows: the 
probability of x in class L meaning x matches the training image IL is zl: and the 
probability of x in class 0, meaning x does not match the training image IL is zO. The 
ABM recognition algorithm only deals with zL If zl is greater than a certain threshold, 
II and X will be a match; otherwise, II and x will not match. 

Step 4, The triplet (x, 1, zl) is conyerted into a doublet (x. zl). A sample output is in 
Figure 5. 

APN Recognition Algorithm is 

1 . An image to be classified is imposed on the Markoy Chain. 

2. This chain will settle on its on its invariant distribution. A distribution function is 
deployed to describe such a distribution. 

3. This distribution function, once obtained, can be used to classify us e d classify images. 
This will produce triplets of image, class, and weight. 

4. Comparing the input-vector and the APN-connection-vector modifies this weight. All 
connection vectors together forms a vector space. A distance, either LI distance or L2 
distance can be defined in this space. The basic idea is the new weight will be directly 
proportional to the old weight and inversely proportional to this distance. The present 
invention covers all functions of obtaining the new weight: 

New weight = f (old weight, distance). 
This will produce a new set of triplets of image, classification, and weight. 

5. These triplets of image, classification, and weight can be viewed as the results of the 
classification process. For the search process, a doublet of image and weight are 
displayed. The second part of the triple is omitted because the search problem has only 
one class. 

Explanation 

Step 1. See Step 1 of the ABM Recognition algorithm above. 
Step 2. See Step 2 of the ABM Recognition algorithm above. 
Step 3. See Step 3 of the ABM Recognition algorithm above. 
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Step 4. The ABM weight is modified bv a formula. 

New weight = f (old weight, distance). 
For example, zl* = zl / (1 + distance). The old triplet (x. K zl) will become a new triplet 

Step 5. The triplet (x, K zD is converted into a doublet (x. zT). A sample output is in 
Figure 5. 

IVI-API (Image Verification and Identification Application Programming 
Interface) 

A typical image matching application structure is: 

• GUI (graphical user interface) Layer 

• DBMS (database management system) Layer 

• rVI-API (image verification and identification API) Layer 

• SPI (Service Provider Interface) Layer 

• OS (Operating System) and Hardware Layer 

The IVI-API is transparent for SPI (Service Provider Interface): the SPI functions will pass right through 
the rVI-APL The SPI can be accessed directly from layers above the IVI-API layer, i.e. the DBMS layer 
or GUI layer. 

There are two main functions in API layer: verify and identify; and there is one main function in the SPI 
layer: capture. 

The two top-level jobs for verification are Enrollment and Verify. The two top-level jobs for 
identification are Enrollment and Identify. The enrollment, in either case, is nothing but setting a few 
parameters; the IVI-API deals with the raw images directly. In this API, there is only one top-level 
function for verifications. Verify; and there is only one top-level function for identifications, Identify. 
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This IVI-API does not have an enrollment process. The enrollment is replaced by setting two parameters: 

• The image in question; 

• The folder of previously stored images. 

This IVI-API does require an image storage structure that should be followed by the applications, so the 
folder of previously stored images can be passed to the verification and identification functions. 
Both the verification path and identification path are parameters, which can be changed by the parameter 
writer functions. The image in question can be stored anywhere in a hard drive. The previously stored 
images must follow the following structure: 

Verification 

The previously stored images must be stored at: 
verification path\ID\. 

Example. Assume: 

1 . The verification path (a parameter) is: 

c:\Attrasofl\verification\ 

2. A set of doublets is: 

Image imagelD 
Ginal.jpg 12001 
Gina2.jpg 12001 
Tiffanyl.jpg 12002 
Tiffany2.jpg 12002 

Then the storage structure is: 

c:\Attrasoft\verification\l 200 l\gina 1 .jpg 
c:\Attrasoft\verification\12001\gina2.jpg 
c:\Attrasoft\verification\l 2002\tiffanyl jpg 
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c:\Attrasoft\verification\12002\tifrany2.jpg 
Identification 

The folder of previously stored images must be stored at: 
identification path\ 

Example. Assume: 

1. The identification path (a parameter) is: 

c:\Attrasoft\identification\ 

2. A set of doublets is: 

Image imagelD 
Ginal.jpg 12001 
Gina2.jpg 12001 
Tiffanyl.jpg 12002 
Tiffany2jpg 12002 
If the number of images is less than 1000, then the storage structure is 

c:\Attrasoft\identification\gina 1 jpg 
c:\Attrasoft\identification\gina2.jpg 
c:\Attrasoft\identification\tiffanyl .jpg 
c:\Attrasoft\identification\tiffany2.jpg 

If the number of images is more than 1000, then the sub-directories should be used: 

c:\Attrasoft\identification\dirOOOO\ginal .jpg 
c:\Attrasoft\identification\dir0000\gina2.jpg 
c:\Attrasoft\identification\dirOOOO\tiffanyl.jpg 
c:\Attrasoft\identification\dir0000\tiffany2.jpg 
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Enrollment 



The enrollment process builds the folder of previously stored images according to the above structure. 
The folder of previously stored images will be a parameter for the AVI layer, called verification directory, 
or identification directory or search directory. There will be a section to address the parameters later. 
Dccauoc the onrollmont moanp poooing poramotoro, the onrollmont io alwayo m%. Enrollment means 
placing images in a folder. For example, if you want to enroll Gina who, has an ID and an image: 

Image imagelD 

Ginal.jpg 12001 
Then you must place the image, ginal .jpg, to the correct folder: 

c:\Attrasoft\verification\l 200 l\gina 1 .jpg 
Later during a verification, if someone claim she is Gina with ID 12001, then the newly captured image 
will be compared with the images in this folder: c:\Attrasoft\verification\12001\. The enrollment rate is 
always 100% because you can always copy an image to a folder. 



1:N Matching 

The following methods (one main fimction and three result readers) are used to perform the Verification 
function: 

int verify(String image, long imagelD); 
long getVerifylDO; 
String getVerifyNameO; 
long getVerifyWeightO; 

A typical process is: 

• Initialize System 

• Capture image 
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• Calculate the template 

• Verify 

However, because "Calculate the template" is not required in this IVI-API; and the system is initialized 
before the verification process started, the process is: 

• Capture 

• Verify 

The captureO functions are provided in SPI, which can be accessed directly by applications. Both the 
image in question and the folder of previously stored images are in the hard drive. The applications then 
pass (String image, long imageBD) to the verifyO function. 

N:N Matching 

The following methods are used to perform the Identification function: 
int identify(String image ); 
Long [] getldentifylDO; 
String [] getldentifyNameO; 
Long getldentifyWeightQ. 

Both the image in question and the folder of previously stored images are in the hard drive. The 
applications then pass (String image) to the identifyO function. 

Parameters 

There are manv parameters used in the algorithm (Figure. 3). such as Blurring, Sensitivity, Lite mal Cut. 

Threshold (External Cut^ This set of parameters together forms a n integer array in the chosen 

programming language. The set of parameters can be accessed via the followin g set- and get-functions: 

Tho o e t of parameters formo an array; 
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Void setParameter( int I, long x); // a[I] 
Long getParameter(int I); // retune a[I] 



Sample Implementation 

We will present three sample implementations based on Figure 2 (3-Layer Architecture). The 
first example has all 3 layers; the second example has only 1 layer; and the third example has 2 
layers. 

There are two CD's labeled "Document, Sample Implementation". The disks contain only three 
ASCII files. Each disk in the duplicate set is identical. The contents of the CD are: 
File Name Type Size Date Description 

ABM4_9 TXT 156,256 05-16-02 Detailed description of LnageFinder 4.9 
ABM5_0 TXT 96,515 05-16-02 Detailed description of PolyApplet 5.0 
ABM5_1 TXT 43,019 05-16-02 Detailed description of Trans Applet 5.1 
These three files will give detailed descriptions of the three sample implementations below. 

Attrasoft ImageFinder 4.9 

A sample Invention Application Software is the Attrasoft ImageFinder 4.9, which has all three 
layers in Figure 2. Figure 3 shows the ImageFinder User Interface using the Present Invention. 
Figure 4 shows a sample Key Input in the hnageFinder software using the Present Invention. 
Figure 5 shows a sample Search Output of the Present Livention. The search output is a list of 
pairs. Figure 6 shows a sample Classification output of the Present Invention. The classification 
output is a Ust of triplets. 

The ASCn file, ABM4_9.TXT, in tiie CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

In addition, two CD's, labeled "Attrasoft hnageFinder 4.9", contain sample implementation 
software. The software can be installed and run to test the proposed algorithm. Note: 
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A. The CD's contain non-ASCII files, such as the installation file and execution files. The 
installation files will install the following executable files to a computer with Microsoft: 
Windows as the operating system: 

• Attrasofl ImageFinder 4.9 for Windows 95/98/ME, execution files; 

• Attrasofl ImageFinder 4.9 for Windows 2000/XP, execution files; 

• Data File for running the software; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 
These five files can also be run fi"om the CD. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the duplicate set is identical. 

D. Contents of the CD. 

Root Directory Contents: 



File Name 


Type 


Size 


Date 




Description 


DISKl 


ID 


5 


01-05-90 


9:31p 


Installation File 


DISKIO 


ID 


5 


01-05-90 


9:31p 


Installation File 


DISKl 1 


ID 


5 


01-05-90 


9:31p 


Installation File 


DISK12 


ID 


5 


01-05-90 


9:31p 


Installation File 


DISKl 3 


ID 


5 


01-05-90 


9:32p 


Installation File 


DISK14 


ID 


5 


01-05-90 


9:32p 


Installation File 


DISK2 


ID 


5 


01-05-90 


9:32p 


Installation File 


DISKS 


ID 


5 


01-05-90 


9:32p 


Installation File 


DISK4 


ID 


5 


01-05-90 


9:33p 


Installation File 


DISKS 


ID 


5 


01-05-90 


9:33p 


Installation File 


DISK6 


ID 


5 


01-05-90 


9:33p 


Installation File 


DISK7 


ID 


5. 


01-05-90 


9:33p 


Installation File 


DISKS 


ID 


5 


01-05-90 


9:34p 


Installation File 


DISK9 


ID 


5 


01-05-90 


9:34p 


Installation File 


SETUP 


EXE 


47,616 


01-05-90 


9:31p 


Installation File 
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SETUP 


INI 


32 


01-05-90 9:3 Ip 


Installation File 


SFTUP 


INS 


147,449 


01-05-90 9:3 Ip 


Installation File 




ISS 


510 


01-05-90 9:3 Ip 


Installation File 


SFTUP 

OX^ X 


PKG 


15,061 


01-05-90 9:3 Ip 


Installation File 


INST32I 


EX_ 


306,666 


01-05-90 9:31p 


Installation File 


ISDEL 


EXE 


8,192 


01-05-90 9:31p 


Installation File 


SETUP 


1 


721,623 


01-05-90 9:31p 


Installation File 


SETUP 


10 


1,454,681 


01-05-90 9:3 Ip 


Installation File 


SETUP 


11 


1,455,574 


01-05-90 9:3 Ip 


Installation File 


SETUP 


12 


1,455,468 


01-05-90 9:3 Ip 


Installation File 


SETUP 


13 


1,454,113 


01-05-90 9:32p 


Installation File 


SETUP 


14 


1,074,165 


01-05-90 9:32p 


Installation File 


SETUP 


2 


1,454,796 


01-05-90 9:32p 


Installation File 


SETUP 


3 


1,456,887 


01-05-90 9:32p 


Installation File 


SETUP 


4 


1,455,245 


01-05-90 9:33p 


Installation File 


SETUP 


5 


1,455,918 


01-05-90 9:33p 


Installation File 


SETUP 


6 


1,455,206 


01-05-90 9:33p 


Installation File 


SETUP 


7 


1,453,720 


01-05-90 9:33p 


Installation File 


SETUP 


8 


1,455,603 


01-05-90 9:34p 


Installation File 


SETUP 


9 


1,456,571 


01-05-90 9:34p 


Installation File 


SETUP 


DLL 


10,752 


01-05-90 9:31p 


Installation File 


oJj* 1 *Jx 


LIB 


196,219 


01-05-90 9:31p 


Installation File 


ABM49 


<DIR> 


06-08-01 l:04p 


Executable File 


USPT072 


<DIR> 


02-28-01 7:15p 


Data File 


USPT074 


<DIR> 


05-21-01 4:33p 


Data File 



E. Interpretation of the files 

Please see Appendix A for the detailed interpretation of the roles of these files. To install the 
software to a Personal Computer using Windows, double click the setup.exe file. 

Attrasoft PolyApplet 5.0 
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A sample Invention Application Software is the PolyApplet 5.0, which only has the Neural Layer 
of this invention. 

The ASCn file, ABM5_0.TXT, in the CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

Attrasoft TransApplet 5.1 



A sample Invention Application Software is the TransApplet 5.1, which has both Neural Layer 
and the Presentation Layer of this invention. 

The ASCn file, ABM5_LTXT, in the CD's labeled "Document, Sample Implementation" will 
give a detailed description. 

In addition, two CD's labeled "Attrasoft TransApplet 5.1" contain sample implementation of the 
software library. Note: 

A. The disks contain only Non-ASCII files. The CD contains the following files: 

• Attrasoft TransApplet 5.1 software library for Windows 95/98/ME/2000/XP, 
COM/DLL file format; 

• Sample Implementation Code; 

• User's Guide in Microsoft Word, and 

• User's Guide in html format. 

B. The Operating System is Windows 95, 98, ME, 2000, and XP. 

C. Each disk in the dupUcate set is identical. 

D. Contents of the CD: 



Root Directory Contents 
File Name Type 



ABM5_1 

CHAP3 

CHAP4 

CHAP5 

CHAP6 

CHAP7 



DOC 
<DIR> 
<DIR> 
<DIR> 
<DIR> 
<DIR> 



Size 
616,448 



Date 

10-21-01 11:28a 
10-19-01 4:31p 
10-19-01 4:31p 
10-19-01 4:31p 
10-19-01 4:31p 
10-19-01 4:32p 



Description 

User's Guide, Word 

Examples 

Examples 

Examples 

Examples 

Examples 
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FBI 

HELP 

OBLIQUE 

README 

TRANS~26 



<DIR> 
<DIR> 
<DIR> 

TXT 567 
DLL 282,112 



06-08-01 l:04p 
10-19-01 4:40p 
06-08-01 l:04p 
10-20-01 10:51a 
10-21-01 11:00a 



Examples 

User's Guide, Word 
Examples 
readme.txt 
COM DLL 



E. Interpretation of the files 

(El) The file labeled "COM DLL" is the COM DLL software library file to be used by users. 
(E2) The directories, labeled "Examples", contain the examples of how to use the COM DLL. 
(E3) The files, labeled "User's Guide, Word" and the directory, "User's Guide, html", contain 
the User's Guide. 
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